Submitted to the Annals of Applied Probability 



ON SOLUTIONS OF KOLMOGOROV'S EQUATIONS FOR JUMP 

MARKOV PROCESSES 

By E.A. Feinberg,*'' 1 " M. Mandava, *'t and A.N. Shiryaev* 
Stony Brook University* 1 and Steklov Mathematical Instituted 

This paper links three ways to construct a nonhomogeneous jump 
Markov process: (i) via a compensator of the random measure of a 
multivariate point process, (ii) as a minimal solution of the backward 
Kolmogorov equation, and (iii) as a minimal solution of the forward 
Kolmogorov equation. The main conclusion of this paper is that, for a 
given measurable Q-function, all these constructions define the same 
transition function. If this function is a transition probability, it is the 
unique solution of the backward and forward Kolmogorov equations. 
For a continuous Q-function, the equality of minimal solutions of the 
backward and forward Kolmogorov equations and the explicit formu- 
lae for these solutions were established by Feller (1940). In particular, 
this paper extends Feller's (1940) results for continuous Q-functions 
to measurable Q-functions. 



1. Introduction. Let (X, 05 (X)) be a standard Borel space, that is, (X, 53(X)) is a 
measurable space for which there exists a measurable injection onto a Borel subset of the 
real line endowed with its Borel cr-field. For a Borel subset E of the extended real line, we 
denote by *B(E) its Borel cr-field. A function P(u,x;t,B), where u,t G M+ :=]0, oo[, u <t, 
x G X, and B G 53 (X), is called a transition function if it takes values in [0, 1] and satisfies 
the following properties: 

(i) for all u,x,t the function P(u,x;t, •) is a measure on (X, 53(X)); 

(ii) for all B the function P(u,x;t, B) is Borel measurable in (u,x,t); 

(iii) P(u, x; t, B) satisfies the Chapman-Kolmogorov equation 

(1) P(u,x;t,B)= / P(s,y;t,B)P(u,x;s,dy), u<s<t. 

Jx 

If P(u, x; t, X) = 1 for all u, x, t, then the transition function P is called a transition 
probability function. 

A stochastic process {X^ : t > 0} with values in X, defined on a probability space 
(0,7",P) with a filtration {P t }t>o, is called Markov if ¥(X t G B \ F u ) = P(X t G B \ X u ), 
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P — a.s for all u, t € with u < t and for all -B € 58(X). And each Markov process has a 
transition function P such that P(Xj £ B | X u ) = X u ; t, 5), P — a.s; see Kuznetsov [18], 
where the equivalence of the two approaches to define a Markov process from Kolmogorov 
[17] is established. If each sample path of the Markov process is a right continuous piecewise 
constant function that has a finite number of discontinuity points on each interval [0, t] for 
t < too, where t^ is the lifetime of the process, then the Markov process is called a jump 
Markov process; Gikhman and Skorokhod [8, p. 187]. In this paper, measurability and Borel 
measurability are used synonymously and all conditional distributions are defined almost 
sure. 

The following function q described below serves as the local characteristic for a Markov 
process defined by its compensator and also for the backward and forward Kolmogorov 
equations. A function q(x,t,B), where x € X, t £ K + , and B £ 58 (X), is called a ex- 
junction if it satisfies the following properties: 

(i) for all x, t the function q(x, t, •) is a signed measure on (X, 58(X)) such that q(x, t, X) 
< and < q(x, t,B\ {x}) < oo for all B £ «8(X); 

(ii) for all B the function q(x,t,B) is measurable in (x,t). 

In addition to properties (i) and (ii), if q(x,t,X) = for all x,t, then the Q-function q 
is called conservative. Note that any Q-function can be transformed into a conservative 
Q-function by adding a state x, to X with q(x,t,{x}) := —q(x,t,X), q(x,t,X) := 0, and 
q(x,t, {x}) := 0, where x € X and t € R+. To simplify the presentation, in this paper 
we always assume that q is conservative. If there is no assumption that q is conservative, 
Remark 4.1 explains how the main formulations change. A Q- function q is called continuous 
if it is continuous in t € M+. 

Let q(x, t) := —q(x, t, {x}). A set B € 58(A) is called q-boundedii sup xgSjtGK q(x, t) < oo 
and the Q-function q is called stable if the set {x} is g-bounded for all x £ X. We make the 
following assumption throughout the paper. 

Assumption 1.1. The Q -function q is stable. 

Let B n := {x £ X : sup tgK+ q(x, i) < n + 1} for all n > 0. Then, f? n "f X as n — >• oo, if 
<7 is stable. Thus, Assumption 1.1 is equivalent to the existence of a sequence of q-bounded 
sets {-B n }n>o such that B n \ X as n — > oo. This way a stable Q-function was defined in 
Feller [7]. 

In this paper, a non-negative solution / in a certain class of solutions of a functional 
equation is called the minimal non-negative solution if for any non-negative solution / 
of this equation from that class /(x) < /(x) for all values of the argument x. Each Q- 
function defines the backward and forward Kolmogorov equations. For a stable continuous 
Q-function, Feller [7] found minimal non-negative solutions of the backward and forward 
Kolmogorov equations, showed that these minimal solutions are equal, and that the minimal 
solution is a transition function. For homogeneous Markov processes, that is Q-functions 
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do not depend on t, Doob [3, 4, Chap. 6] provided an explicit construction for multiple 
transition functions satisfying the backward Kolmogorov equation, Kendall [13], Kendall 
and Reuter [14], and Reuter [19] gave examples with non-unique solutions to Kolmogorov 
equations, and Reuter [19] provided necessary and sufficient conditions for their uniqueness; 
see also Chen et al. [2]. Ye, Guo, and Hernandez-Lerma [20] investigated a countable state 
problem with Borel measurable Q- functions. 

A conservative Q-function can be used to construct a predictable random measure. Ac- 
cording to Jacod [12, Theorem 3.6], an initial state distribution and a predictable random 
measure define uniquely a multivariate point process. Theorem 2.2 given below states that 
the stochastic process associated with the multivariate point process defined by a sta- 
ble conservative Q function q and an initial state distribution is a jump Markov process. 
Theorem 2.2 also describes its transition function P. By using the methods introduced by 
Feller [7] for continuous Q-functions, in Theorems 3.1, 4.1 we show that the transition func- 
tion P satisfies the backward and forward Kolmogorov equations written for the Q-function 
q. In addition, P is the minimal non-negative solution of the backward and forward Kol- 
mogorov equations and, if this transition function is a transition probability, then it is the 
unique non-negative solution of the backward and forward Kolmogorov equations that is a 
measure with values in [0, 1]; Theorems 3.2, 4.3. In particular, this paper extends Feller's [7] 
results from continuous to measurable Q- functions. Though the uniqueness property of the 
transition probability function P was stated in Feller [7, Erratum], the proof is missing. 

Our interest in this study is motivated by its applications to continuous-time Markov 
decision processes. Here we mention two of them: 

(i) For a countable state space, each Markov policy along with a given initial state distri- 
bution defines a Markov process with the transition function being the minimal solution of 
the forward Kolmogorov equation; Guo and Hernandez-Lerma [9, Section 2.2]. An arbitrary 
policy defines a multivariate point process via the compensator of its random measure; Ki- 
taev [15], Kitaev and Rykov [16, Section 4.6], Feinberg [5, 6], Guo and Piunovskiy [10]. The 
results of this paper imply that for Markov policies these two definitions are equivalent for 
problems with Borel state spaces. 

(ii) Feller's [7] results are broadly used in the literature on continuous-time Markov 
decision processes to define Markov processes corresponding to Markov policies; see Guo 
and Hernandez-Lerma [9] and references therein. The application of Feller's results typically 
leads to the unnecessary assumption that decisions depend continuously on time; see, e.g., 
[11, Definition 2.2]. For countable state problems the results in Ye et al. [20] removed 
the necessity to assume this continuity. The results of the current paper imply that this 
continuity assumption is unnecessary for Markov decision processes with Borel state spaces. 

2. Relation between Jump Markov Processes and Q-functions. The main goals 
of this section are to show that an initial state distribution and a (stable) Q-function q define 
a jump Markov process and to construct its transition function. 
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Let 

^oo ^ X be an isolated point adjoined to the space X. Denote X — X U {xqo} 
and R+ = ]0,oo]. Consider the Borel <7-field 53(X) = cr(53(X), {x^}) on X, which is the 
minimal cr-field containing 53 (X) and {xoo}- Let (X x ]R_|_)°° be the set of all sequences 
(xo,ti,xi,t2,X2, ■ ■ •) with x n £ X and t n+ \ £ R + for all n > 0. This set is endowed with 
the a- field generated by the products of the Borel a- fields 55 (X) and 23 

Denote by the subset of all sequences oj = (xo,ti,xi,t2,X2, ■ ■ ■) from (X x ]R + )°° such 
that: (i) xq £ X; (ii) if t n < oo, then t n < t n+ i and x n E X, and, if t n = oo, then 
t n+ i = i n and x n = Xqo, for all n > 1. Observe that f2 is a measurable subset of (X x ]R + )°°. 
Consider the measurable space (iljj 7 ), where J 7 is the a- field of the measurable subsets 
of O. Then, x n (oj) = x n and t n +i(o;) = i n +lj n > 0, are random variables defined on 
the measurable space (fl, T). Let to := 0, too(^) := hm t n (oj), oo £ CI, and for all t > 

n— >oo 

let Ji := o-(53(X),ft), where ft := <t(J{x„ E B}I{t n < s} : n > 1,0 < s < t, B £ 
53 (X)). Throughout this paper, we omit oj whenever possible and also follow the standard 
convention that x oo = 0. 

For a given Q-function q, consider the random measure v on (R + x X, 53(M+) x 53(X)) 
defined by 

(2) u{uj;]0,t],B)= [ y2l{t n <s<t n+1 }q{x n ,s,B\{x n })ds, t £ R + , B £ 53(X). 
Jo n>0 

Note that v({i},X) = u{[t oa ,oo[,X) = and (2) can be rearranged as 

' n— 1 „/ , , _+ 



/ q(x m , t m + s, i? \ {x m })(is 

m=CT 

pt—tn \ 

+ y q(x n ,t n + s,.B \ {z n })dsj . 

As the expression in the parentheses on the right hand side of (3) is an Tt n -measurable 
process for each B £ 53(X), it follows from Jacod [12, Lemma 3.3] that the process 
{^(]0, t], B) : t £ M+} is predictable. Therefore, the measure v is a predictable random 
measure. According to Jacod [12, Theorem 3.6], the predictable random measure v defined 
in (2) and a probability measure \x on X define a unique probability measure P on (Q, J-) 
such that W(xq £ B) = (J,(B), B £ 53(X), and v is the compensator of the random measure 
of the multivariate point process (t n ,x n ) n >i defined by the triplet (fi, T , P). 
Consider the process {X t : t > 0}, 

}x n + /{too < t}x 

oo i 



n>0 



defined on (fi, J 7 , P). We abbreviate the process {Xf : t > 0} as X. The main result of this 
section, Theorem 2.2, shows that the process X is a jump Markov process and provides its 
transition function. 
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For an ^-measurable stopping time r, let N(r) : = max{n = 0, 1, . . . : r > t n }. Since 
N(t) = oo and X T = {xoo} when r > t^, we follow the convention that too+i = oo and 
Xqo+i = %oo- Denote by G T (u);-, •) and H t (oj;-) respectively the regular conditional laws 
of (ij\r(r)+ij x N(t)+i) an d ijvM+i with respect to T T \ H t {uj\ •) = G t (uj; -,X). In particular, 
Gt n (uj; •, •) and H tn (u; •), where n = 0, 1, . . . , denote the conditional laws of (t n+ i, x n+ i) 
and t n+ i with respect to Tt n ■ We remark that the notations Gt n and H tn correspond to the 
notations G n and H n in Jacod [12, p. 241]. 

Lemma 2.1. For all u,t G R+, u <t, 

(5) H u ([t, oo}) = e-J^CX-.-)*, jV(u) < oo, 

(6) G u (dt,B) = e-^ q{Xu ' s)ds q(X u ,t,B\{X u })dt, B G <B(X), X(u) < oo. 

PROOF. According to Jacod [12, Proposition 3.1], for all t G R + , B G 05 (X), and n = 
0,1,... 

(7) u(dt, B) = Gu f ' g) , i n <t<t n+1 . 

In particular, for B = X, from (7) and from the property that x n+ \ G X when t n+ \ < oo, 

G tn {dt,X) H tn (dt) 

v{dt,X) = 77 = 77—777 77, t n <t< t n+ i. 

Ht n ([t, 00]) H tn {[t,oo\) 

This equality implies that z/(dt, X) is the hazard rate function corresponding to the distri- 
bution Ht n when t n < t < £ n +l- Therefore, 

(8) = e -"(]*«.*].^{*»<*<*"+i} ) teR + ,t>t n . 
From (2) and (8), for all t G K + , 

(9) ^ n ([t,oo]) = e-4^)^ ) 4>tn; 
and from (2), (7), and (9), for all i € K+, B G 25(X), 

(10) G tn (dt,B) = e-^ q{Xn > s)ds q(x n ,t,B\{x n })dt 1 t > t n . 
To compute G u , observe that for all u,t £ R+, u <t, and G 55 (X), 

(11) G u (dt,B) =P(% (u)+ i G [M + d*[,ZiV(u)+i G S I J- tt ) 

n>0 

= P(t n+ i e [M + , ^n+i G £ I Ju, JV(u) = n)/{X(u) = 
n>0 
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where the first equality follows from the definition of G u , the second equality holds because 
{N(u) = 00} U {N(u) = n} n= o,i,... is an J^-measurable partition of Q and xtv( u )+i = ^00 ^ 
X when N(u) = 00, and the third equality follows from N(u) = n and from {N(u) = n} G 
T u . Observe that for any random variable Z on (f2, T) 

(12) P(Z I T u , N(u) = n)I{N{u) = ?i} = P(Z | T tn , N(u) = n)I{N(u) = n} 

= P(Z I T tn ,t n < u,t n+1 > u)I{N{u) =n} = P(Z \ F tn ,t n+l > u)I{N{u) = n} 
F{Z,t n+1 >u\T tn ) 



-I{N(u) = n}, 



where the first equality follows from Bremaud [1, Theorem T32, p. 308], the second equality 
holds because {t n < u,t n+ i > u} = {N(u) = n}, the third equality holds because {t n < 
u} G J-j n , and the last one follows from the definition of conditional probabilities. 

Let Z = {t n+1 G [t,t + dt[,x n+1 G B}, where t G M+ , B G %>(X). Then (11) and (12) 
imply 

/in\ n / j, m \- F(t n+ i G [t,t + dt[,x n+1 G g I jy 

= £ e ~ Jl '" ,{ '""'' d '^n,t,B \ {a;„})dt = 



n>0 



g(X u ,t, J B\{X u })dt, 



where the first equality holds because {i n +i G [t, i + dt[,t n+ \ > u} = {t n+ i G [t,t + dt[} 
when t > u, the second equality follows from (9) and (10), and the last equality holds 
since x n = X u when N(u) = n. For all u,t G M+, tt < t, it follows from the property that 
xtv(u)+i £ -X" when tjy(tt)+i < 00 an d from (13) that H u ([t, 00]) satisfies (5). □ 

Following Feller [7, p. 501], for x £X, u,t £ R+, u < t, and B G *B(X), define 

(14) P(°) (it, x; t, B) = I{x G B}e~ Si q ^ ds , 
and for n > 1 define 

(15) p( n \u,x;t,B)= f [ e-Su9( x > e ) de q(x,s,dy)P( n - 1 \s,y;t,B)ds. 

Ju JX\{x} 

Set 

00 

(16) P(u,x;t,B) := ^ P (n) (u, x; t, S). 

n=0 
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Observe that P is a transition function. For stable continuous Q-functions, Feller [7, Theo- 
rems 2, 5] proved that (a) for fixed u, x, t the function P(u, x; t, •) is a measure on (X, 03 (X)) 
such that < P(u, x;t,-) < 1, and (b) for all u, x, t, B the function P(u, x; t, B) satisfies the 
Chapman-Kolmogorov equation (1). The proofs remain correct for measurable Q-functions 
q. The measurability of P(u, x; t, B) in u, x, t for all B G 05 (X) is straightforward from the 
definitions (14), (15), and (16). Therefore, the function P satisfies properties (i)-(iii) from 
the definition of a transition function. 

Theorem 2.2. For a given initial state distribution and for a stable Q-function q, the 
process X defined in (4) is a jump Markov process with the transition function P. 

Proof. Observe that the sample paths of the process X are right-continuous piecewise- 
constant functions that have finite number of discontinuities on each interval [0, t] for t < too- 
Thus if, for all u,ie R + , u < t, and B G <B(X), 



then the process X is a jump Markov process with the transition function P. To prove (17), 
we first establish by induction that for all n = 0, 1, . . . , u, t € R+, u < t, and B € 95 (X) 



where iVwi := N(t) — N(u) when u < too and -/Vwi := oo when u > too- Equation (18) 
holds for n = because for u < too 



(19) P(X t G B, JV ]Ujt] = 0\T U ) = P(X M G B, tjv (u)+1 >t\T u ) 

= /{X„ G B}B u (]t, oo]) = /{X„ G Bje-tizMds = _p( )( Ui X M ; t, B), 



where the first equality holds because the corresponding events coincide, the second equality 
holds because {X n G B} G T u and from the definition of H u , the third equality is correct 
because of (5), and the last equality is (14). 

For some n > 0, assume that (18) holds. Then for u < too 



(17) 



P(X t G B [ T u ) = P(X t G B | X u ) = P(u, X u ;t, B) 



u < t. 



(18) 



P(X t G B, iV ]tt>t] = n | J„) = B( n )( U , X u ; t, B) 



u <t, 



(20) P(X t GB,iV M 



n + 1 | F u ) 





n I 



)G„(<it W '( u ) +1 , <fejv(u)+i) 




Q(X n , s, dy)e- lM6Bp(n) ^ y . ^ B)ds = p(n+l) ( „ ? ^ . ^ j g ) 
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where the first equality holds since Nj u>t j = 1 + ^]t N(u)+1 ,t] f° r ^]u,t] > 1 an d since E(E(Z | 
D)) = E(Z) for any random variable Z and any cr-field 2), the second equality holds since 
0" *AT(u)+i! ^jv(u)+i) = ■? r t JVW+1 , the third equality follows from (6) and (18), and the 
last equality is (15). Equality (18) is proved. 



Observe that for u,t G M+, u < t, B G 58 (X), 

(21) P(X f eB\T u ) = P(X t G B | J"„)/{« < too} + P(X f G B | -F u )/{n > too} 

= £ P(X t G B, JV M = n | T u )I{u <t oa } = J2 P {n) («, < W 

n>0 n>0 

= X u ; t, B)I{u < too} = X u ; t, B)/{X U G X}, 



where the first equality holds since {{u < too}) {u > too}} is a partition of and {u < 
too}) > t^} S Fu-, the second equality holds since Xj £ 1 implies t < too, the third 
equality follows from (18), the fourth equality follows from (16), and the last one holds 
since {u < too} = {X u G X}. As follows from (21), the function P(X t G B | J 7 ^) is cr(X u )- 
measurable. Thus, 



where the second equality holds because cr(X u ) C J 7 ^; see e.g. Bremaud [1, p. 280]. Thus, 



3. Backward Kolmogorov equation. In this section, we show that the transition 
function P defined in (16) is the minimal non-negative solution to the backward Kolmogorov 
equation. For a continuous Q-function q, relevant results were established by Feller [7, 
Theorems 2, 3]. 

Theorem 3.1. The function P(u,x;t, B) satisfies the following properties: 
(i) P(u,x;t, B) is for fixed x,t,B an absolutely continuous function in u and satisfies 
uniformly in B G 58 (X) the boundary condition 



(ii) For all x,t,B, the function P(u, x;t, B) satisfies for almost every u < t the backward 
Kolmogorov equation 



(22) 



P(X t G B | 3F U ) = P(P(X t eB\JF u )\ Xu) = P(X t G B \ X u ) 



(17) follows from (21) and (22). 



□ 



(23) 



lim P(u, x; t, B) = I{x G B}. 



(24) 




Proof, (i) For all x G X, u,t G K+, u < t, and B G 5B(X), 
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(25) P(u, x- t,B) = ^T P (n) (n, x- 1, B) 

n=0 

= I{x G B}e~ & q( ^ x ' s)ds + V / e~ ^ 9(a: ' 0)de / g(x, s, dy)P {n - l) (s, y; t, 5)ds 

„=i ■/« 

= /{a; G P}e~ /« q ^ ds + e~ St i^ e ) dd / q( x , s, dy) V P^" 1 ) (s, y; t, B)ds 

Ju Jx\{x} ^ 

= /{a; G B}e~ St + / e" £ ?(a: ' 0)de / g(x, s, dj/)P(s, y; t, J3)ds, 

where the first equality is (16), the second equality follows from (14) and (15), the third 
equality is obtained by interchanging the integral and sum, and the last one follows from 
(16). For fixed x,t,B, equation (25) implies that P(u,x;t,B) is the sum of two absolutely 
continuous functions in u. Thus, P(u, x; t, B) is for fixed x, t, B an absolutely continuous 
function in u. 

Observe that P^(u, x; t, B) < P(u,x;t,B) < 1 for all n > 0, x G X, u, t G R+, u < t, 
and B G <B(X). Then from (15), 

/•t 

(26) pW(u,i;t,5)< e-^ q ^ do q(x,s)d8, n > 1. 



This inequality and (14) imply that, for any stable Q-function q, 

(27) lim P» (u, x; t, B) = for all n > 1 and lim p(°) («, 2; t, B) = /{x G P} 

u— u—>t~ 

uniformly with respect to B. Thus, (16) and (27) imply (23). 

(ii) Since an absolutely continuous real-valued function is differentiable almost everywhere 
on its domain, for all x, t, B the function P(u, x; t, B) is differentiable in u almost everywhere 
on ]0,t[. By differentiating (25), for almost every u < t, 



(28) ^-P(u, x; t, B) = I{x G B}e~ St *M ds q(x, u) - [ q(x, u, dy)P(u, y; t, B) 
du Jx\{x} 

+ f |-e- S: ^ de [ q(x, s, dy)P(s, y; t, B)ds 

Ju OU Jx\{x} 

= I{x G B}e~ St i( x ^ ds q( x , u) - [ q(x, u, dy)P(u, y; t, B) 

Jx\{x} 

+ f e~ q{x,u) I q(x,s,dy)P(s,y;t,B)ds. 

Ju Jx\{x} 

In view of (25), the sum of the first and the last terms in the last expression of (28) is equal 
to the first term on the right-hand side of (24). □ 
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As shown in Feller [7, Theorem 2], for a stable continuous Q-function q, the transition 
function P satisfies the backward Kolmogorov equation for all u, while Theorem 3.1(h) 
states that this equation holds for almost every u. This difference in formulations takes 
place because the continuity of the Q-function q and the finiteness of each integrand in the 
last expression of (25) guarantee the existence of the derivative J^P(u, x; t, B) for all u. 

Definition 3.1. A function P with the same domain as P is a solution of the backward 
Kolmogorov equation (24) if the function P satisfies the properties stated in Theorem 3.1. 

The next theorem describes the minimal and uniqueness properties of the solution P of 
the backward Kolmogorov equation (24). 

Theorem 3.2. The function P is the minimal non-negative solution of the backward 
Kolmogorov equation (24). Also, if P is a transition probability function (that is, P(u,x;t,X) 
= 1 for all u,x,t in the domain of P), then P is the unique non-negative solution of the 
backward Kolmogorov equation (24) that is a measure on (X,*B(X)) for fixed u,x,t with 
u < t and takes values in [0, 1] . 

Proof. The proof of minimality is similar to the proof of Theorem 3 in Feller [7]. We 
provide it here for completeness. Let P* with the same domain as P be a non-negative 
solution of the backward Kolmogorov equation (24). Integrating (24) from u to t and by 
using the boundary condition (23), 

(29) P*(u,x;t,B) = I{x € B}e~ q ^ ds 

+ f f e-f>^ de q(x,s,dy)P*(s,y;t,B)ds. 

Ju JX\{x} 

Since the last term of (29) is non-negative, 

(30) P* (u, x; t , B) > I{x G B}e~ q{x ' s)ds = P (0) (u,x;t,B), 

where the last equality is (14). For all u,x,t,B with u < t, assume P*(u,x;t, B) > 

n _ 

£ p( m \u,x;t,B) for some n > 0. Then from (29) 

m=0 

P*{u,x;t,B) > I{x e B}e~ti q ( x ' s)ds 

+ / e -f:^ de q(x,s,dy)y^P^\s,y;t,B)ds 

Ju Jx\{x} m=Q 

n n+1 

= P(°) (u,x;t,B)+J2 P [m+1) (u,x;t,B) = J2 P M {u,x;t,B), 

m=0 m=0 
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n _ 

where the first equality follows from the assumption that P* (u, x;t,B) > ^ p( m ) (u,x;t,B) 

m=0 

for all u,x,t,B with u < t, the second equality follows from (14) and (15), and the third 

n _ 

equality is straightforward. Thus, by induction, P*(u,x;t,B) > ^ p( m \u,x;t,B) for all 

m=0 

n > 0, x G X, u,t G R + , u < t, and B G 0S(X), which implies that P*(u,x;t,B) > 
P(u, x; t, B) for all u, x, t, B. 

To prove the second part of the theorem, let the solution P* be a measure on (X, 03 (X)) 
for fixed u,x,t and with values in [0,1]. Assume that P*(u,x;t, B) ^ P(u,x;t, B) for at 
least one tuple (u,x,t, B). Then, 

P*(u, x; t, X) = P*{u, x; t, B) + P*{u, x; t, B c ) 

> P(u, x; t, B) + P(u, x; t, B°) = P(u, x;t,X) = l, 

where the inequality holds because P*(u,x,t, •) > P(u,x,t, •) for all u,x,t. Since P* takes 
values in [0,1], the assumption that P*(u,x;t,B) ^ P(u, x;t, B) for atleast one tuple 
(u,x,t,B) leads to a contradiction. □ 

4. Forward Kolmogorov equation. For the forward Kolmogorov equation, this sec- 
tion provides the results similar to the results on backward Kolmogorov equation in Sec- 
tion 3. 

Theorem 4.1. The function P(u,x;t, B) satisfies the following properties: 

(i) P(u,x;t, B) is for fixed u,x,B an absolutely continuous function in t and satisfies 
uniformly in B G 03 (X) the boundary condition 

(31) lim P(u, x; t, B) = I{x G B}. 

t^>-u+ 

(ii) For all u,x, and q-bounded sets B, the function P(u,x;t, B) satisfies for almost every 
t > u the forward Kolmogorov equation 

(32) -|p(u, x;t,B) = -J q{y, t)P(u, x; t, dy) + J q(y, t, B \ {y})P{u, x; t, dy). 

PROOF, (i) For all x G X,u,t G M+, u < t, and B G 03 (X), equation (25) implies that 
the function P(u, x; t, B) is absolutely continuous in t for fixed u,x,B. Also, equations (14) 
and (26) imply that, for any stable Q- function q, 

(33) lim P (n) (u, x; t,B)=0 for all n > 1 and lim P (0) (n, x; t, B) = I{x G B} 

t—>-u+ t—>u+ 

uniformly with respect to B. Thus, (16) and (33) imply (31). 
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(ii) Consider the following non-negative function denned on the domain of P by 

(34) n(«, x; t, B) = [ q(x,u,dy)e-fui(y> s ) ds . 

Jb\{x} 

According to Feller [7, Theorem 4], the function p( n \u, x; t, B), n > 1, satisfies the recursion 

(35) P {n) {u,x;t,B)= [ [ U(s,y;t,B)P {n - 1) {u,x;s,dy)ds. 

Ju Jx 

Though the function P^ n \u,x;t, B) is defined for continuous Q-functions in Feller [7], the 
proof given there is correct for Borel Q-functions. From (14), (16), and (35), 

oo 

(36) P{u, x; t,B) = ^T P (n) (u, x; t, B) 

n=0 

t 00 r l r 

= J{ieB}e"J , . ? ( I ' s > ds + V / U(s,y;t,B)P ( - n - 1 \u,x;s,dy)ds 

n=1 Ju JX 

= J{x£B}e-i ?M *+ / / U(s,y;t,B)y2P {n ~ 1 Hu,x;s,dy)ds 

Ju JX n=1 

= J{i£B}e-£« M *+ I ( U(s,y;t,B)P(u,x;s,dy)ds. 

Ju Jx 

Since P(u,x;t,B) is an absolutely continuous function in t for fixed u,x,B, the derivative 
^P(u, x;t, B) exists for almost every t &]u, oo[. By differentiating (36), for almost every 
t > u, 

(37) ^P(n, x- 1, B) = -I{x G B}e~ & q ^ ds q(x, t) 

+ J n(t,y;t,B)P(u,x;t,dy) + J ^ J U(s,y;t, B)P(u,x;s,dy)ds 
By differentiating (34) with respect to i, for all ^-bounded sets B € B(X), 

(38) JWs;i,B) = / q ( x , u ,dy)-^-e-I^(y^ = - [ q (y,t)U(u,x;t,dy). 
dt Jb\{x} ot J B 

Combining (37) and (38) and observing that II(i, y; t, B) = q(y, t, B\{y}), for all g-bounded 
sets B, 

(39) J^P(u, x- 1, B) = -I{x G B}e~ £ iM ds q ( x , t) 
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q{y,t,B\{y})P(u,x;t,dy) - / / q(z,t)n(s,y;t,dz)P(u,x;s,dy)ds, 
x Ju Jx JB 

for almost every t > u. By substituting P(u, x; t, dz) in the left-hand side of the following 
equality with the final expression in (36), 

(40) / q(z, t)P(u, x- t, dz) = I{x G B}e~ & q ^ ds q(x, t) 
JB 

+ q(z,t)U(s,y;t,dz)P(u,x;s,dy)ds. 

Ju JX JB 

Formulae (39) and (40) imply statement (ii) of the theorem. □ 

Corollary 4.2 (Feller [7, Equation (37)]). For all x G X,u,t £ R + , u < t, and 
q-bounded sets B G *B(X), the function P(u,x;t, B) defined in (16) satisfies 

(41) P(u, x; t, B) = I{x G B} 

+ ds q(y,s,B\{y})P(u,x;s,dy) - ds q(y,s)P(u,x;s,dy). 

Ju JX Ju JB 

PROOF. Integrating (32) from u to t and by using the boundary condition (31), we get 
(41). Thus, by Theorem 4.1, for all x G X, u, t G R+, u < t, and (/-bounded sets B G *B(X), 
the function P(u,x;t, B) satisfies (41). □ 

Definition 4.1. A function P with the same domain as P is a solution of the forward 
Kolmogorov equation (32) if the function P satisfies the properties stated in Theorem 4-1- 

Following the proof of Theorem 3.2, we establish the minimal and uniqueness properties of 
the solution P of the forward Kolmogorov equation (32) in theorem 4.3. We remark that the 
function P is the minimal solution of (32) on a restricted domain (u, t G M+,u <t,i6l, 
and g-bounded sets B G <B(X)). 

Theorem 4.3. The function P(u,x;t,B), being restricted to q-bounded sets B, is the 
minimal non-negative solution of the forward Kolmogorov equation (32). Also, if P is a 
transition probability function (that is, P(u,x;t,X) = 1 for all u,x,t in the domain of P), 
then P is the unique non-negative solution of the forward Kolmogorov equation (32) that is 
a measure on (X, ( B(X)) for fixed u,x,t with u <t and takes values in [0, 1]. 

Proof. Let P* defined on the same domain as P be a non- negative solution of the 
forward Kolmogorov equation (32). Integrating (32) from u to t and by using the boundary 
condition (31), for all x G X, u,t G M+ with u <t, and (/-bounded sets B G ^(X), 



(42) P*(u,x;t,B) = I{x G B}e~ ^ x ^ ds + [ ds I U(s, y; t, B)P*{u, x; 

Ju JX 



s,dy), 
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for all (/-bounded sets B. Since the last term of (42) is non-negative, 

(43) P*{u,x;t,B)>I{x£ B}e' q{x ' s)ds = P (0) (u,x\t,B), 

where the last equality is (14). For all x,u,t with u < t and (/-bounded sets B, assume 

n _ 

P*(u,x;t,B) > £ p( m \u,x;t,B) for some n > 0. Then from (42) 

m=0 

t r f r n 

P* (u, x; t, B) > I{x G B}e~ J« i( x ' s ) ds + / d s / n(s, y; i, B) ^ P< m )(u, x; s, dy) 

Ju Jx m=0 

n n+1 

= P ( -°\u,x;t,B) + Y,P (m+1) (u,x;t,B) = ^ P^{u, x; t, B). 



m=0 m=0 



Thus, by induction, P*(u,x;t,B) > ^ =0 P( m '(M,x;f,e) for all n > 0, x G X, u, i € M+ 
with u <t, and (/-bounded sets £> G 2$P0, which implies that P*(u, x; t, B) > P(u, x; t, B) 
for all u,x,t with u < t, and for all (/-bounded sets B. 

To prove the uniqueness property of P, let the solution P* be a measure on (X, 03 (X)) 
for fixed u,x,t with u < t and with values in [0,1]. It follows from statement (i) of the 
theorem that for all B G 05 (X) 

(44) P*(u, x; t, B) = lim P*(u, x;t,BD B n ) > lim P(u, x;t,BD B n ) = P(u, x; t, B), 

n— >oo n— >oo 

where {-B n }n>o is an increasing sequence of (/-bounded sets such that B n "f X as n t °0) 
whose existence is guaranteed by Assumption 1.1. If P(u,x;t,X) = 1 for all u,x,t, then 
the uniqueness of P within the set of solutions to the forward Kolmogorov equation that 
take values in [0, 1] and that are measures on (X, 05(X)) for fixed u,x,t with u < t follows 
from the minimality of P (44) and from the same arguments as in the proof of uniqueness 
in Theorem 3.2. □ 



Remark 4. 1 . The results of this paper can be extended to non-conservative Q-functions. 
As mentioned in section 1 , any non-conservative Q- function q can be transformed into a con- 
servative Q-function by adding a state x to X with q(x,t, {x}) := —q(x,t,X), q(x,t,X) := 
0, and q(x,t,{x}) := 0, where x G X and t G M+. According to Theorem 2.2, there is a 
transition function P of a jump Markov process with the state space X = X U {x}, and 
this process is determined by the initial state distribution and by the compensator defined 
by the modified Q-function. The proofs of the results of sections 3 and 4 do not use the 
assumption that the Q-function q is conservative. Therefore, these results remain valid for 
non-conservative Q-functions. However, the validity of the condition P(u, x; t, X) = 1 for 
all x, u, t with u < t in Theorems 3.2 and 4.3 is possible only if q(x,t,X) = almost 
everywhere in t for each x G X. Thus, in fact, q is conservative, if P(u, x; t, X) = 1 for all x, 
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u, t with u < t. It is also easy to see that the minimal solutions of both the backward and 
forward Kolmogorov equations are equal to P(u, x; t, B), when i£l and B E *B(X), where 
the transition function P is described in the previous paragraph for a broader domain. 

Remark 4.2. In this paper, transition functions P(u,x;t,B) and P(u,x;t,B) are de- 
fined for u > 0. All the results of Sections 3 and 4 hold for u > with the same proofs. 
When x is the initial state of the process X, the results of Section 2 also hold for u > 0. 
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